Framework tests verify behavior instead of implementation text by MariusStorhaug · Pull Request #551 · PSModule/Process-PSModule

Marius Storhaug (MariusStorhaug) · 2026-11-04T11:43:27Z

Focused framework checks now verify release-note content and authored-page transfer instead of fixed log sentences, documentation phrases, or a private helper parameter inventory. Version outputs, failure handling, recovery, WhatIf behavior, module fixtures, and the shipped validation suites remain covered.

This is a scoped test/CI cleanup. Production scripts, consumer inputs, documentation, coverage targets, permissions, and release labels are unchanged. The PR stays in draft because the existing producer-conformance and publication-evidence holds are not resolved here.

Technical details

Scope and immutable identities

Repository: PSModule/Process-PSModule. Default-branch base: 64ffb1d87f86bd5d487891fdb140b78385a00531. Reviewed head: 7a00cbb9cd72baddf14d990019cf5da3c9c3600e. Seven small ordinary commits change five files. The worktree is clean and the head is pushed. No open PR branch is a source.

Changed file	Result
`.github/actions/Release-PSModule/tests/Release-PSModule.WhatIf.Tests.ps1`	Replace a diagnostic-phrase assertion with real Git commit selection and notes-file content observed at an intercepted GitHub write boundary. Use two different commits, verify the selected target, reject unexpected WhatIf calls, and clean up the shim after each case.
`.github/actions/Publish-PSModule/tests/Publish-PSModule.Helpers.Tests.ps1`	Remove the helper parameter-inventory guard. All six SemVer output/invalid-input cases remain.
`.github/workflows/Workflow-Test-Default.yml`	Compare the generated root landing page with its nonempty authored input instead of three fixed phrases.
`.github/workflows/Workflow-Test-WithManifest.yml`	Apply the same input/output contract to the manifest fixture.
`.github/workflows/Test-Actions.yml`	Remove tautological configuration echoes and the later configuration reset. Run the configured shuffle/seed/start markers, with explicit serial file execution because the suites share process environment variables. Keep discovery and failure gates.

Complete audit inventory

PSModule/Process-PSModule#550 contains the complete source-bound classification of every assertion and invoked test/discovery entry point, including counts, real entry points, expectations, and policy reasons. Counts below describe the audited source; no fixed repository-inventory assertion is added.

Surface	Disposition and contract
Get-PSModuleSettings (13 cases)	Retain real event routing, exact merged-commit association, controlled tree/compare outputs, and incomplete-response failures.
Resolve-PSModuleVersion (45 cases)	Retain real JSON normalization, version baselines/composition, next versions, context, and label-conflict failures. Version strings and controlled response counts are functional expectations, not snapshots. Existing anonymous Gallery reads do not independently prove the Gallery service contract.
Publish helpers (6 cases)	Retain stable/prerelease SemVer, whitespace handling, and invalid empty input. Remove only the parameter-absence guard.
Publish recovery (3 cases)	Retain real script branching for simulated existing/absent packages and lookup failures. Unsafe vendor writes remain intercepted.
Release (4 cases)	Retain action outputs, dry-run target, and simulated resume inputs. Rewrite manual dispatch to observe notes from the selected real commit rather than log prose.
update-index (1 case)	Retain actual catalog/page generation and links from controlled repository metadata. Owned remote metadata boundaries remain simulated.
Shipped SourceCode suite (10 definitions, data-driven links)	Retain all documented static-analysis IDs. This is a supported validation product, not an edit-detection assertion against its own source.
Shipped Module suite (7 definitions, data-driven types)	Retain actual import/manifest/version-folder validation, type registration, and OnRemove cleanup. A verified pre-existing discovery/oracle gap is tracked separately in #553.
Both containers and configurations	Retain the actual Test-PSModule/Invoke-Pester discovery, report, and coverage wiring.
Both fixture module suites (5 cases each)	Retain real exported greeting-function and SolarSystems variable behavior.
Both fixture Environment suites (2 cases each), TestData.Assertions, BeforeAll/AfterAll	Retain exact controlled environment transport expectations in tests and phases. Fixture values/lengths are input data, not source snapshots.
Two generated-index workflow checks	Rewrite six frozen phrase checks as an authored-input transfer contract. Different headings/prose are accepted; lost content and empty inputs fail.
Other workflow assertions and quality gates	Retain test-result/coverage, schema, lint, build, setup/teardown, and missing/unexecuted-result controls. Existing fixture skip settings are unchanged. Guidance benchmarks/runtime examples and site JavaScript add no undiscovered assertion runner.

Validation and evidence limits

Check	Result and boundary
Exact-head action CI	Run 41827042422: 72 passed, 0 failed, 0 skipped, Pester 6.2.0 within the unchanged `[6.1.0,7.0.0)` range. The baseline had 73 cases; the sole removed case was the parameter inventory.
Local action loop	The actual workflow test step passes with installed Pester 6.1.0, shuffled order, and serial files. Pester rejects an empty selection rather than reporting a pass.
Release fault checks	Wrong notes fail. Reading HEAD instead of CommitSha fails with the different newer commit's text. A diagnostic-only wording change passes. All experimental production edits were restored.
Documentation contract	The real generator and the exact two workflow assertion scripts pass 12 controlled checks: original/changed authored wording accepted for both layouts, lost content rejected, empty input rejected. Hashes compare input and output files, not a stored snapshot or generator source.
Retained SourceCode validator	A valid isolated source fixture passes 10/10; an injected OutNull defect produces the intended single failure.
Retained Module validator	A real source-bound CI artifact passes 26/26. Removing registration in an isolated copy produces 21 intended failures. A comment-driven false green is documented in #553; the shipped suite is not removed or weakened.
Parallel-isolation probe	The actual Pester 6.1 parallel runner uses the same process for both controlled workers; one worker observes the other's environment value and fails. Serial execution preserves the baseline's actual mode without suppressing a test.
Script/workflow lint	Clean-profile, Pester-aware PSScriptAnalyzer 1.25.0: 0 findings. Actionlint 1.7.12: pass. Exact-head LF blobs with the pinned CI YAML rules: 0 errors, 23 warnings. No lint configuration or suppression is changed.
Manual Zizmor	The same four existing broad-permission findings occur on base and head. They are tracked in #552; no permission change is authorized in this PR.
Exact-head repository CI	52 successful checks, 7 existing conditional skips, no failed or pending checks. Both fixture workflows, root-index checks, test-result/coverage collection, lint, and CodeQL succeed.

The seven conditional skips are the two fixture publication jobs of each kind, the default fixture's repository lint, and the manifest fixture's source test/lint jobs. These are not new passes. Live Gallery publication, real GitHub release writes, publication provenance, and safety-owner handback remain unexecuted here. Test doubles prove local caller handling only; they do not prove an upstream service, authentication, or safe publication ownership.

The compiled-fixture check first detected a missing ThreadJob 1.x dependency. Only that declared dependency was restored to isolated scratch storage; no global installation changed. An early Windows scratch-build attempt hit formatter line-ending limits, so the successful retained-Module evidence uses the real Ubuntu CI artifact from run 41824863677/source dbd162f0001b73cfa03207dd18f0ce607adc6114. The compiler and shipped validator are unchanged between that source and the final head. An inherited-shell analyzer run also emitted an intermittent Context/Get-Context resolution warning; baseline/candidate rechecks returned 0/0 and clean-profile analysis returned 0, without a suppression.

Review disposition

Author self-review covers the complete scoped diff. The first independent automated review identified two valid gaps: the selected commit was also HEAD, and parallel execution could race on process environment. Both were fixed, exercised with deliberate counterexamples, replied to, and resolved.

Copilot review 5768041720 is APPROVED for the exact final head 7a00cbb9cd72baddf14d990019cf5da3c9c3600e; its complete summary reports no active issues and both prior findings resolved. No new inline finding remains. This is an automated approval, not a claimed human approval. The author-side reply records are not counted as independent reviews.

Standards and framework alignment

Changed surface	Standards checked	Framework documentation checked	Result
Release/Publish action tests	MSX Testing, PowerShell Testing, naming, functions, error handling, natural language	Existing action contracts/help and version/release behavior	Fixed in this PR: real behavior replaces incidental phrase/signature checks; simulation limits are explicit.
Generated-index workflow assertions	MSX Testing, YAML, GitHub Actions	Actual Build-Docs route and Build-PSModuleDocumentation authored-Markdown transfer	Fixed in this PR: complete input/output preservation replaces frozen phrases.
Test-Actions runner	MSX Testing, PowerShell Testing, GitHub Actions	Writing module tests and current Pester/action contract	Fixed in this PR: real configured execution; serial files retain isolation.
Existing fixture-workflow permissions	GitHub Actions least privilege	Actual called workflow permission contract	Exception: #552. No permission or required-control change is included.
Retained shipped type validation	MSX Testing independent oracle and meaningful examined cases	Existing Module validation contract	Exception: #553. Separate supported-contract repair, not a compiler/publisher rewrite here.

Relevant canonical policy files were unchanged when refreshed through MSXOrg/docs@1b0d4893c8fefabe604d04393068a8ff6f848082. Existing guidance already describes the scoped runtime contracts; no documentation/native-help path needs a cleanup edit.

The Task's audit, focused implementation, local verification, CI, and exact-head automated review steps are complete. The issue-convergence search covered functional-test alignment, snapshot/change-detection terms, touched components, and the two new follow-ups. Only #550 is fully addressed. #184 and all unrelated/held proposals remain open and unchanged.

Producer hold

The actual repository release route is .github/workflows/Release.yml calling PSModule/Release-GHRepository@750abed0ecab02394554b610dc3aa191e75e3248 (v2.0.3). Its immutable released interface uses legacy labels; no adapter to the canonical release-decision contract is verified. Publication/source baselines for this versioned workflow producer are not established by this scoped diff.

The draft therefore makes no release decision, release-label, final version, consumer-publication coverage, or ready-for-merge claim. The existing producer-conformance/publication-evidence holds remain intact. No merge, auto-merge, label removal, publication, permission change, or held-branch integration was performed. Original-tested-artifact and safety-owner decisions are unchanged.

Related references

Resolves [functional-test-alignment] Prove framework behavior without freezing implementation text #550
References Scope fixture workflow permissions to their calling jobs #552
References Module validation hides missing type registration when an exporter comment changes #553
References MSX Testing standard

Copilot

🟡 Changes recommended

Parallel test suites can race on shared environment variables, and the release-notes test cannot detect whether commit-message selection ignores CommitSha.

2 open issues

What changed in this PR

This PR advances the framework’s test-alignment audit by checking observable release behavior and generated documentation rather than fixed wording or implementation details. Production scripts and consumer inputs are unchanged.

Changes:

Verify release-note contents and target commits at an intercepted GitHub command boundary, and reject GitHub calls in WhatIf cases.
Compare generated Functions indexes with authored fixtures using file hashes.
Remove implementation-focused assertions and preserve the configured Pester options through execution.

File	Description
`.github/workflows/Workflow-Test-WithManifest.yml`	Verify authored Markdown is preserved in manifest-based builds.
`.github/workflows/Workflow-Test-Default.yml`	Verify authored Markdown is preserved in default builds.
`.github/workflows/Test-Actions.yml`	Remove configuration self-checks and retain Pester execution settings.
`.github/actions/Release-PSModule/tests/Release-PSModule.WhatIf.Tests.ps1`	Assert release notes and target arguments using a real Git fixture and intercepted GitHub calls.
`.github/actions/Publish-PSModule/tests/Publish-PSModule.Helpers.Tests.ps1`	Remove a parameter-introspection assertion.

_{Review effort: Balanced}

Copilot

🟢 Approved

No active review issues are identified for the changes aligning tests with observable behavior.

No open issues 🎉

2 resolved since last review

_{Review effort: Balanced}

Check release notes instead of diagnostic wording

ab0219e

Marius Storhaug (MariusStorhaug) self-assigned this Nov 4, 2026

Marius Storhaug (MariusStorhaug) added 4 commits November 4, 2026 12:55

Use an isolated Git commit for release-notes coverage

57fb3b0

Keep version-output checks without freezing helper parameters

61ba657

Check authored documentation transfer without fixed prose

ed4af52

Run action tests with the configured Pester controls

dbd162f

Marius Storhaug (MariusStorhaug) requested a balanced review from Copilot November 4, 2026 11:58

Copilot started reviewing on behalf of Marius Storhaug (MariusStorhaug) November 4, 2026 11:58

View session

Marius Storhaug (MariusStorhaug) mentioned this pull request Nov 4, 2026

[functional-test-alignment] Prove framework behavior without freezing implementation text #550

Open

7 tasks done

Copilot AI reviewed Nov 4, 2026

View reviewed changes

Comment thread .github/actions/Release-PSModule/tests/Release-PSModule.WhatIf.Tests.ps1 Resolved

Comment thread .github/workflows/Test-Actions.yml Resolved

This was referenced Nov 4, 2026

Scope fixture workflow permissions to their calling jobs #552

Open

Module validation hides missing type registration when an exporter comment changes #553

Open

Marius Storhaug (MariusStorhaug) added 2 commits November 4, 2026 13:10

Distinguish the dispatched commit from repository HEAD

40d2a1d

Keep shared-environment test files in serial execution

7a00cbb

Marius Storhaug (MariusStorhaug) requested a balanced review from Copilot November 4, 2026 12:11

Copilot started reviewing on behalf of Marius Storhaug (MariusStorhaug) November 4, 2026 12:12

View session

Copilot AI approved these changes Nov 4, 2026

View reviewed changes

Navigation Menu

Repository navigation

Framework tests verify behavior instead of implementation text - #551

Framework tests verify behavior instead of implementation text#551
Marius Storhaug (MariusStorhaug) wants to merge 7 commits into
mainfrom
framework-test-alignment

Marius Storhaug (MariusStorhaug) commented Nov 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Marius Storhaug (MariusStorhaug) commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scope and immutable identities

Complete audit inventory

Validation and evidence limits

Review disposition

Standards and framework alignment

Producer hold

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approved

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Marius Storhaug (MariusStorhaug) commented Nov 4, 2026 •

edited

Loading